Endeca Assembler : How to debug MDEX Engine query results?

The MDEX Engine provides several methods for understanding why certain results were returned for a query so that you can determine how to tune search features to provide the desired results.
 

There are four main query debugging features:

1) Why Match 

2) Why Rank

3) Why Precedence Rule Fired

4) Word Interpretation 

How to Enable query debugging features?

In ATG, use below component to return debugging information from the assembler.

 /dyn/admin/nucleus/atg/endeca/assembler/cartridge/manager/MdexRequestBroker 

$constructor.param[1].value=true

In addition, Word Interpretation must be enabled via the --wordInterp Dgraph flag.
 

The relevant configuration for the individual cartridge handlers in the CRS reference application is shown below:

1) Results List — Why Match, Why Rank

/atg/endeca/assembler/cartridge/handler/config/ResultsListConfig/

whyMatchEnabled=true

whyRankEnabled=true

2) Refinement Menu — Why Precedence Rule Fired 

/atg/endeca/assembler/cartridge/handler/config/RefinementMenuConfig/

whyPrecedenceRuleFired=true

3) Search Adjustments — Word Interpretation 

/atg/endeca/assembler/cartridge/handler/config/SearchAdjustmentsConfig/

showWordInterp=true

Endeca query types - Endeca Navigation Query, Endeca Record Search, Endeca Dimension Search, Endeca Aggregated Record Query

Basic Query	Returned Object(type)	URL Param	Query Example
Navigation Query	Navigation	N	/endeca_jspref/controller.jsp?enePort=15000&eneHost=localhost&N=4294966668
Record Query	ERec	R	/endeca_jspref/controller.jsp?R=sku790435010109597
Dimension Search Query	DimensionSearchResult	D	/endeca_jspref/controller.jsp?D=red+wine
Aggregated Record Query	AggrERec	A, An, Au	/endeca_jspref/controller.jsp?Au=product.repositoryId&A=sku056009010128567&An=4294947584

Endeca ENEConnectionException - How to Enable backwards compatibility, so that the Dgraph can communicate with previous versions of the Presentation API

ERROR MESSAGE in Endeca JSP reference application: ENEConnectionException com.endeca.navigation.ENEConnectionException: Error establishing connection to retrieve Navigation Engine request

Enable backwards compatibility, so that the Dgraph can communicate with previous versions of the Presentation API. In addition to the currently supported version of the Presentation API, the following previous full versions are supported: 6.0.x, 5.1.x, 5.0.x and 4.8.x. Therefore, the value for <api-version> must be one of the following:

• 601 for all 6.0.x versions of the API.
• 510 or 500 for all corresponding versions of the API, 5.1.x and 5.0.x.
• 480 for the 4.8.x versions of the API, including the Perl API. 

<dgraph-defaults>
    <properties>
     ...
    <directories>
     ...
    </directories>
    <args>
       <arg>--back_compat</arg> 
       <arg>601</arg> 
       ...
        </args>
    <startup-timeout>120</startup-timeout>
  </dgraph-defaults>

Note: Starting with version 6, the Endeca Presentation API is part of the Platform Services package. For the version of the Platform Services that is compatible with the current version of the MDEX Engine.

Endeca Record Aggregation - Rollup key and Benefits

Endeca Record Aggregation (Rollup) allows developers to do below two things:         
       1)  On Listing page, results can be rolled up based on a shared property or dimension value
       2)  On Product Details Page,  all aggregated records can be requested

Benefits of Record Aggregation (Rollup):
        1)  Avoid multiple records that are essentially same
              Ex: On search item with 5 sizes and 5 colors will return 25 results instead of 1 record
        2)  Organize a record list based on a common value
               Ex: A list of unique SKU’s grouped by Product
        3)  Avoid losing precision from consolidation
               Ex: Each unique SKU has a Unique inventory state 

How to define Rollup key?  
Refer -  http://ravihonakamble.blogspot.com/2015/07/oracle-commerce-11x-how-to-define.html

Endeca aggregated records in JSP reference application:

Assembler Code to fetch Endeca Experience Manager and Endeca GuidedSearch content - Endeca Assembler API

Code snippet to invoke Endeca Experience Manager Content:

<dsp:droplet name="/atg/endeca/assembler/droplet/InvokeAssembler">
            <dsp:param name="includePath" value="/pages/browse" />
            <dsp:oparam name="output">
               <dsp:getvalueof var="contentItem" vartype="com.endeca.infront.assembler.ContentItem"         param="contentItem" />
            </dsp:oparam>
</dsp:droplet>

Code snippet to invoke Endeca Guided Search content:

 <dsp:droplet name="/atg/endeca/assembler/droplet/InvokeAssembler">
            <dsp:param name="includePath" value="/services/guidedsearch" />
            <dsp:oparam name="output">
               <dsp:getvalueof var="contentItem" vartype="com.endeca.infront.assembler.ContentItem" param="contentItem" />
            </dsp:oparam>
  </dsp:droplet>

Input Parameters

The InvokeAssembler servlet bean has two input parameters, includePath and contentCollection.

includePath

Use the includePath parameter for a page request.

For example, to assemble content for a browse page, specify /pages/browse for the includePath (passing in a /browse path will not match because it is missing the /pages prefix).

contentCollection

Use the contentCollection parameter for a content collection request.

For example,  Search Box Auto Suggest Content. InvokeAssembler parses the contentCollection into a ContentSlotConfig component.

The includePath and contentCollection parameters are mutually exclusive but one of them must be passed when using the InvokeAssembler servlet bean.

ruleLimit

This optional parameter is used in conjunction with the contentCollection parameter to specify the number of items that should be returned from the specified content collection.

Output Parameters

The InvokeAssembler servlet bean has one output parameter, contentItem. This parameter contains the root ContentItem returned by the Assembler. If this content item is empty, the request was not an Assembler request.

Assembler API - Oracle Endeca

Oracle Endeca Commerce 3.1 includes the Endeca Assembler, a new query API and framework that replaces the Java and .NET Content Assembler from previous releases.

Benefits are :

• Supports multi-channel natively by serializing responses to multiple formats (JSON, XML and Java objects)
• Centralizes application business logic in one place using cartridge handlers 
• Supports multiple deployment modes as an in-process Java library or as a RESTful service
• Provides out-of-the-box, extensible core cartridges for querying the MDEX Engine
• Provides extensible architecture for querying other source systems, allowing business users to assemble the full digital experience using Experience Manager
• Consolidates run-time back end queries when possible
• Includes URL optimization for SEO
• Returns view-ready response models

Endeca integration API libraries in ATG DAF module - ATG Endeca Integration

ATG 10.1.1 include the following Endeca libraries.

/ATG10.1.1/DAF/Endeca/Assembler/lib/endeca_assembler-3.1.0.jar
/ATG10.1.1/DAF/Endeca/Assembler/lib/endeca_navigation-6.3.0.jar
/ATG10.1.1/DAF/Endeca/Index/lib/custom

ATG 10.1.2 includes
/ATG10.1.2/DAF/Endeca/Assembler/lib/endeca_assembler-3.1.1.jar
/ATG10.1.2/DAF/Endeca/Assembler/lib/endeca_navigation-6.4.0.jar
/ATG10.1.2/DAF/Endeca/Index/lib/custom
/ATG/ATG10.1.2/DAF/Endeca/Index/lib/custom311

ATG 10.2 includes 
/ATG10.2/DAF/Endeca/Assembler/lib/endeca_assembler-3.1.2.jar
/ATG10.2/DAF/Endeca/Assembler/lib/endeca_navigation-6.4.1.jar
/ATG10.2/DAF/Endeca/Index/lib/custom311

Endeca Dimension search Performance Tuning - Best Practices

Performance of dimension search directly depends on number of dimension values and size of the resulting set of matching dimension values. If Dimension values returned are more than 1000 then it will cause big performance hit.
Compound dimension search is more expensive than non-compound request. if you are not using compound dimension feature then disable it. Here is the link on how to disable it:
http://ravihonakamble.blogspot.com/2015/07/endeca-few-items-on-dgrpah-side-to.html
 
In both the cases, You can limit the results by using advanced dimension search parameters.
Ex: Di parameter to specify the specific dimension or list of Dimension value Id's. 

D=avengers&Di=11378
D= Dimension Query Search Term
Di= One or more ID's of dimensions to search against

Endeca Search and performance Impact - Record Search|Wildcard Search|Boolean Search|Phrase Search

Record Search:
Record search is an indexed feature, each property enabled for record search increases the size of the Dgraph process. Only properties that are needed by an application for record searching should be configured.
Use Endeca Set Selection feature :  http://ravihonakamble.blogspot.com/2015/06/endeca-select-feature-aka-set-selection.html

Wildcard Search:
If wildcard search is enabled in the MDEX Engine (even if it is not used by the users), it increases the time and disk space required for indexing. Therefore, consider first the business requirements for your Endeca application to decide whether you need to use wildcard search. 
Recommendations:
1) Avoid wildcard searches with one non-wildcarded character, such as a* , since they are more expensive to process
2) Parse the queries to calculate their search term length to avoid very low information queries, such  as "a*". Avoid MDEX Engine wildcarding queries that contain fewer than 3 non-wildcarded characters. 
3) Remove all non-searchable characters from each wildcard query before issuing it to the MDEX Engine 
4) Exclude wildcard queries with quoted phrase searches.
FYI 
If search queries contain only wildcards and punctuation, such as *.* , the MDEX Engine rejects them for performance reasons and returns no results. 

Boolean Search:
 The performance of Boolean search is a function of the number of terms and operators in the query and also the number of records associated with each term in the query. As the number of records increases and as the number of terms and operators increase, queries become more expensive. If you notice unexpected behavior while using Boolean search, use the Dgraph-v flag when starting the Dgraph. This flag prints detailed output to stderr describing the running Boolean query process.

Phrase Search:

The cost of phrase search operations depends mostly on how frequently the query words appear in the data and the number of words in the phrase. You can improve performance of phrase search by limiting the number of words in a phrase with the --phrase_max <num> flag for the Dgraph. 

Using this flag improves performance of text search with phrases. The default number is 10. If the maximum number of words in a phrase is exceeded, the phrase is truncated to the maximum word count and a warning is logged.

OOTB - Endeca Record Filter Builder Components

The ATG-Endeca integration includes several record filter builder components. Here are few which are most commonly used:

 1) A filter that restricts the set of records returned to only those associated with specified sites. 

It uses product.siteId attribute.

 /atg/endeca/assembler/cartridge/manager/filter/SiteFilterBuilder 

 2) A filter that restricts the set of records returned to only those in the current language. 

It uses product.language attribute.

 /atg/endeca/assembler/cartridge/manager/filter/LanguageFilterBuilder

3) A filter that restricts the set of records returned to only those associated with the appropriate catalogs.

It uses product.catalogId attribute.

 /atg/endeca/assembler/cartridge/manager/filter/CatalogFilterBuilder

 To enable a specific record filter builder component, you add it to the recordFilterBuilders property of the  

/atg/endeca/assembler/cartridge/manager/NavigationStateBuilder component. 

recordFilterBuilders+=\
/atg/endeca/assembler/cartridge/manager/filter/CatalogFilterBuilder,

/atg/endeca/assembler/cartridge/manager/filter/SiteFilterBuilder 

Endeca Select feature aka set Selection - Controlling Endeca Record Values

The Select feature allows you to select specific keys (Endeca properties and/or dimensions) from the data so that only a subset of values will be transferred for Endeca records in a query result set.

This functionality prevents the transferring of unneeded properties and dimension values when they will not be used by the front-end Web application.

It therefore makes the application more efficient because the unneeded data does not take up network bandwidth and memory on the application server.

There are two possible scenarios:
1)  If you are making independent Endeca queries then you can use below API:

// Create a query

ENEQuery usq = new UrlENEQuery(request.getQueryString(),"UTF-8");

// Create an empty selection list

FieldList fieldList = new FieldList();

// Add an Endeca property to the list

fieldList .addField("P_DisplayName");

// Add an Endeca dimension to the list

fieldList .addField("P_RepositroyId");

// Add the selection list to the query

usq.setSelection(fieldList );

// Make the MDEX Engine query

ENEQueryResults qr = nec.query(usq);

2)If you are using ATG-Endeca integration then use below component and add only required fields.

 /atg/endeca/assembler/cartridge/handler/config/ResultsListConfig.properties

 fieldNames=\
  P_DisplayName,\
  P_RepositroyId